In many application scenarios, one or more applications produce messages that are intended to be processed exactly once by multiple consumers, including other applications or users. Each such message is deemed processed if, and only if, all of the consumers have processed the message. Further, each consumer must process the message in exactly the same order that is determined by the application, for example, on a first in-first out basis, or on a prescribed priority basis.
One approach for managing the delivery of messages to multiple consumers is to provide a separate message queue for each consumer. When a message is to be enqueued, or otherwise stored in a queue for subsequent use by one or more consumers, for each consumer that is to read the message, the message is added to that consumer's individual message queue.
When a message is to be dequeued, or otherwise read by a consumer, the message is returned to the consumer from the consumer's respective message queue. The message is then deleted from the consumer's message queue, and the dequeueing process is ended.
In this method, each consumer has its own individual message queue. Thus, a first consumer has a first message queue, a second consumer has a second message queue, and so on.
While this approach creates no concurrency problems, as each consumer accesses its own individual message queue to read the messages destined for it, there are design features that make this method generally unfavorable. First, this method requires considerable amounts of memory, for each message must be replicated at least as many times as there are consumers to read it. As a portion of messages in any system can be expected to be relatively large, the demands on memory imposed by this method may be difficult or even unmanageable for many systems.
In addition, this method requires complexity in message management, due to the replication of a single message for use by many different consumers. If, after a message is enqueued to the various consumer message queues, it is to be modified, altered, or otherwise acted on in any way, the system will have to replicate the change as many times as there are instances of the message. With many consumers, this maintenance chore can be expected to adversely impact the overall system performance.
Another approach for managing the delivery of messages to multiple consumers is to use one message queue for all the consumers, but lock each message queue entry that is currently being accessed. In this way, when a first consumer accesses a particular message in the message queue, no other consumer can access the message until the first consumer has completed its transaction and unlocked the message queue entry.
In this method, a reference count is used to keep track of when a message in the queue has been read by all the consumers and can be deleted from the message queue. The reference count may be a field in the message queue, or it may be a variable in a separate table, array or list. When a message is enqueued to the message queue, a corresponding reference count for that message is set to the maximum number of consumers that are to read the message. Thereafter, each time a consumer reads the message, the associated reference count is decremented. When the reference count becomes equal to zero, it is assumed that all consumers that need to access the message have done so, and the message is deleted from the message queue.
While this method resolves the problem of the large memory usage posed by the previously described method, by using generally one copy of a message for all consumers, it creates concurrency problems between the consumers. As all consumers must process the messages in the message queue in the same order, the messages in the message queue become hotspots, with one or more consumers possibly queued behind a first consumer who has gained access to a first message. This can significantly affect access time to the messages by other consumers. Depending on various system and environmental factors, such as the number of consumers accessing a message, the time required to complete a transaction involving a read on the message, and a consumer's relative position waiting to access the message in the queue, consumers may encounter significant delays to access one or more of the queued messages.
Yet another method for handling the delivery of messages to multiple consumers is to enqueue a message to a single message queue that all relevant consumers can access, and store an associated reference count in the queue for use in subsequently deleting the message from the queue. The global message queue has a field of message entries, for storing each of the enqueued messages. The global message queue also has a second field, associated with the field of message entries, that contains a reference count for each of the enqueued messages. When a message is added to the queue, the associated reference count is set to the maximum number of consumers who are to read it.
In this method, unlike the approach described immediately above, a message queue entry is not locked when a user reads the respective message in the queue. Instead, at some point during the transaction in which a consumer reads a message, the reference count associated with the message is decremented. A deletion process determines if the reference count for any particular enqueued message is set to zero. If the reference count associated with an enqueued message is zero, it is assumed that all consumers that are to read the associated message have read it; the message is then deleted from the queue. If, however, the associated reference count for an enqueued message is not equal to zero, indicating that not all consumers that are to read the message have, the message is not deleted from the message queue.
This method has the memory-savings advantage that generally only one copy of a message is enqueued and used by all the consumers that are to read it. However, this method does not resolve concurrency problems between consumers accessing the same message at relatively the same time. While this method does not require an enqueued message to be locked when a consumer reads it from the queue, the reference count in the queue associated with the message is locked when it is updated, i.e., decremented, and when it is referenced during the deletion processing. Thus, the reference count field becomes a hotspot. The ultimate effect is that all but a first consumer must wait for the first consumer that has accessed a reference count to complete its transaction before they can perform theirs. The transaction that consumes a particular enqueued message can take an arbitrary amount of time to complete, taking into consideration commit or rollback scenarios involving the message processing. As the reference count associated with the particular enqueued message must remain locked until the transaction is completed, it can remain locked for an arbitrary amount of time, negatively impacting other consumers' processing and overall system performance.
Still another approach involves the storage of messages in a single queue, in which message sequence numbers and low water marks are used to manage the access of enqueued messages by multiple consumers. According to this method, when a new message is added to the message queue, it is logically associated with a monotonically increasing sequence number. Each consumer is associated with a low water mark value which indicates the next message in the queue the consumer is to read. Thus, implicitly, a consumer's low water mark value indicates all the enqueued messages a consumer has already read, as the enqueued messages are stored in monotonically increasing order in the message queue.
When a consumer reads an enqueued message, the message in the queue with the sequence number corresponding to the consumer's current low water mark value is accessed. The consumer's low water mark value is then incremented. If, however, the consumer's low water mark value is greater than the maximum message sequence number, there are no enqueued messages for the consumer to read.
In this approach, when a deletion process is activated, the lowest low water mark value for all the consumers is identified. The deletion process then deletes all the messages in the queue with associated sequence numbers that are less than the identified lowest low water mark value.
An advantage of this method is that it reduces the memory usage required to provide one message to many consumers, as generally one copy of a message is read by all the respective consumers. Moreover, the concurrency problems associated with other methods do not exist with this method as there is no need to write to, and, thus, lock a portion of, the message queue during a consumer's read message transaction. However, this method has other serious limitations that circumscribe its scope of applicability.
First, with this method, messages cannot be prioritized; they must be processed on a first in-first out basis. This is because the sequence numbers associated with enqueued messages must be in sequential order for the associated read message and deletion processing to operate correctly. Another drawback is that each consumer that accesses the message queue, even those that access the message queue only once, must be provided an associated low water mark value. Thereafter, that consumer's low water mark values must be maintained forever, to allow effective deletion processing. As the number of consumers accessing the message queue increases, the number of low water mark values to be maintained and to be accessed by the deletion process increases correspondingly. Conceptually, the number of consumers who ever access the message queue can grow infinitely large. Thus, the processing associated with this method can likely become unduly complex, maintaining and managing a very large number of low water mark values.
Thus, it is desirable to have a reliable message delivery system and process that ensures that the same messages can be viewed by multiple consumers, in a prescribed order, without causing degradation of any consumer's processing performance because of other consumers' access to the same message(s). Too, it is desirable to have a reliable message delivery system that requires a minimum amount of memory to maintain. Further, it is desirable for the message delivery system to handle message queue deletions in an optimum manner.